Labeled Memory Networks for Online Model Adaptation

We propose an alternative design of memory augmented neural networks (MANNs) called Labeled Memory Networks (LMNs) suited for tasks requiring fast adaptation of batch-trained classification models. LMNs have two key differences with existing MANNs. First, LMNs organize the memory with the discrete class label as the primary key unlike the existing practice of key being a real vector derived from the input. Second, LMNs treat memory as a second boosted stage following a regular neural network thereby allowing the memory and the primary network to play complementary roles. Unlike existing MANNs that write to memory for every instance and use LRU based memory replacement, LMNs write only for instances with non-zero loss and use label-based memory replacement. These properties make them particularly suited for classification tasks requiring fast adaptation and memorizing rare events. We demonstrate significant accuracy gains on three such tasks: online sequence prediction, life-long learning of rare events, and few-shot learning.